Recover a diverged repository from a source replica - #1333

Motivation:

Once a repository has diverged across replicas and been made read-only, there
is no way to repair it: the ZooKeeper replication log cannot move a git ref
backward and old entries are trimmed away. An operator needs a way to
designate one replica's repository as the source of truth and make every other
replica converge to it.

Modifications:

- Add RecoverRepositoryRequestCommand, a trigger that is applied as a no-op on
  every replica; the source replica will react to it in the replication layer.
- Add RecoverRepositoryCommand, a self-contained command that carries the
  commits to replay. Each ReplayCommit holds the original author, timestamp,
  summary, detail, markup, a self-contained change set and the expected commit
  ID of the revision, so every replica converges to identical commit SHAs.
- Dispatch both commands in StandaloneCommandExecutor.
- Add RepositoryManager.recoverRepository(): force-move the git head back to
  the reset revision (new GitRepository.doForceRefUpdate() that allows
  non-fast-forward updates), reopen the repository so the commit-id database
  rebuilds from the git HEAD, replay the carried commits while asserting each
  commit ID, and swap the repository instance. Any mismatch rolls back to the
  original instance. A repository that is already converged (the source or a
  healthy replica) is left untouched, which also makes recovery idempotent.

Result:

- The building blocks for repository recovery are in place: applying a
  recovery command resets a diverged replica onto the source history and
  verifies convergence commit-by-commit. Not yet reachable from any endpoint;
  the ZooKeeper coordination and the admin API follow.

Motivation:

The recovery commands and the git reset-and-replay core existed but were not
reachable: nothing coordinated who builds the recovery payload in a ZooKeeper
cluster, and no endpoint let an operator start a recovery.

Modifications:

- Coordinate the recovery over the replication log, originated by the source:
  - Add RecoveryPayloadBuilder, which builds a self-contained
    RecoverRepositoryCommand from the local storage of the source replica via
    the new RepositoryManager.buildRecoveryPayload() (per-revision history,
    self-contained changes and the expected commit ID).
  - When a replayed RecoverRepositoryRequestCommand names this replica as the
    source, ZooKeeperCommandExecutor reacts off the replay thread by building
    and originating the recovery command. A RecoverRepositoryCommand failure
    already trips only that repository's read-only scope because the failure
    handling maps any RepositoryCommand to its project/repository.
- Add POST /api/v1/projects/{p}/repos/{r}/recover (system administrators
  only): rejected unless the server runs in replicated (ZooKeeper) mode, the
  source server ID is part of the cluster, the repository is file-based, and
  the repository is read-only - so no new commit can race the payload build.
  If the request lands on the source replica the recovery is originated
  directly (status COMPLETED); otherwise a request command asks the source
  over the replication log (status REQUESTED).
- Add GET /api/v1/replicas returning the cluster roster from the static
  replication configuration, marking the replica that served the request; an
  empty list in standalone mode, which also gates the UI.

Result:

- An operator can repair a diverged repository with a single admin API call.
  Verified end-to-end on a 3-replica cluster with a truly diverged replica:
  both the direct and the request-and-react paths converge every replica back
  to the source history, the repository stays read-only until made writable,
  and subsequent pushes replicate cleanly.

Motivation:

Repository recovery was only reachable through the admin REST API. An operator
repairing a diverged repository needs to see which replicas exist, pick the
source of truth and start the recovery from the web UI, with a strong guard
against running such a destructive action by accident.

Modifications:

- Add a "Repository Recovery" settings tab, shown only to system
  administrators and only in replicated (ZooKeeper) mode, gated on the new
  GET /api/v1/replicas endpoint; visiting the page in standalone mode explains
  why the feature is unavailable.
- Add a recovery form with project/repository auto-complete, a source-server
  select fed by the replica list (marking the serving replica), and a start
  revision input.
- Guard the destructive action with a type-to-confirm modal that spells out
  what is rewritten and that commits existing only on non-source replicas are
  discarded.
- Show the outcome inline: a failure is rendered inside the modal (which stays
  open) and a success as a persistent alert under the form, so the result does
  not depend on the transient toast alone. Toasts are still dispatched.
- Add getReplicas / recoverRepository to the API slice.
- Add a jest test covering the enablement rule, the type-to-confirm gate, the
  request payload, the inline failure and the persistent success message, and
  extend the repo-status Playwright spec with the standalone-mode gating.
- Add ReplicatedShiroCentralDogmaTestServer (and a
  runTestReplicatedShiroServer task) that runs the Shiro test server in
  single-replica ZooKeeper mode, so replication-only UI can be developed and
  verified against a real replicated server.

Result:

- A system administrator can run a repository recovery from the web UI with
  clear, persistent feedback. Verified live against a replicated test server:
  tab gating, form flow, the read-only precondition rejection shown inline and
  a completed recovery reporting the converged head revision.

Motivation:

A five-lens review of the recovery feature surfaced one correctness gap and
several robustness, observability and consistency issues:
- The read-only precondition was checked only at request time. A command that
  made the repository writable again (and any push after it) could land in the
  replication log between the payload build and the recovery apply; applying
  the reset would then silently discard those commits on every replica.
- A reader blocked on the old repository instance during the in-place reset
  could wake up after the swap and read the rewritten commit-id database
  through the stale instance.
- A failure in the rollback path could skip the write-lock release, and a
  synchronous executor failure in the source's recovery reaction was dropped
  without a log.

Modifications:

- Re-verify the read-only precondition when RecoverRepositoryCommand is
  applied, using only the replicated repository/project scope
  (RepoStatusManager.isRepoOrProjectReadOnly) so every replica decides
  identically; on the originating replica the recovery aborts before the
  command enters the log.
- Fail fast a reader that was blocked on the old repository instance:
  mark the instance close-pending (new GitRepository.markClosePending) before
  releasing the write lock, and decouple close scheduling from the
  close-pending flag so the later close() still releases the resources.
- Guard the rollback path so a failing close of the partially recovered
  instance cannot leak the old repository's write lock, log synchronous
  failures of the recovery reaction, reject a reset revision above the local
  head with an actionable message, and report a single-commit repository as
  having nothing to replay.
- Type the recover endpoint response (RecoverRepositoryResponse) instead of an
  untyped map, and document the recovery constraints (force-push races,
  best-effort REQUESTED path, rolling upgrades) on the endpoint.
- Web UI: clear the previous outcome when a new recovery is configured, label
  the confirmation input for screen readers, and reword the REQUESTED result
  to say the recovery is asynchronous and best-effort instead of implying the
  Repository Status page can verify convergence.
- Tests: replay coverage now includes multi-file, JSON and removal commits and
  builds payloads through the production buildRecoveryPayload (plus bounds
  checks); JSON round-trips for the new commands; cluster tests for the
  replica roster, an out-of-range start revision, idempotent double recovery
  and the writable-abort re-check; a SettingView test pinning the tab-order
  invariant that keeps the hidden Recovery tab from misaligning the highlight.

Result:

- A recovery can no longer discard commits that raced it, stale reads through
  the swapped-out repository instance fail fast, and the operator-facing
  contract (typed response, honest REQUESTED wording, documented constraints)
  matches what the system actually guarantees.

Motivation:

An adversarial review round (five reviewers, each attacking one invariant of
the recovery feature) confirmed three exploitable gaps:
- The apply-time read-only re-check reads an in-memory cache that is loaded by
  an asynchronously registered listener. A replica restarting with the
  recovery still in its replication backlog could replay it before the cache
  loaded, answer "writable" while the rest of the cluster answered
  "read-only", abort the recovery and poison-skip the log entry - ending up
  silently diverged behind an already-read-only flag.
- A repository whose replay range contains transformer-written entries (the
  per-project internal dogma/meta repositories, e.g. metadata.json) can never
  converge: transformers store JSON without the trailing-newline text
  normalization, while the recovery payload and its replay both normalize, so
  the replayed commit ID always mismatches and recovery aborts after doing the
  work.
- The recovery payload is unbounded. It crosses the replication log as a
  single entry and is materialized in memory by every replica, so recovering a
  large repository from an early revision could exhaust heaps cluster-wide,
  stall replication while replicas replay it, and bloat ZooKeeper.

Modifications:

- Make the read-only re-check trustworthy on a cold cache: a read-only answer
  from the cache is authoritative, but a writable answer is re-verified
  against the replicated status files, which reflect the replication-log
  position on every replica.
- Reject recovery of internal repositories at the endpoint and hide them from
  the recovery form; mention the not-byte-reproducible cause in the commit-ID
  mismatch error.
- Cap the recovery payload (10,000 commits / 64 MiB estimated content) with a
  clear message to recover from a later revision.
- Refresh the replica roster whenever the recovery page or form mounts instead
  of trusting a session-old cache, guard against re-entrant submissions, and
  drop a stale comment about a removed gc lock.
- Tests: a cold-cache manager must answer read-only from the stored status
  (reproduces the restart race); payload-cap boundaries; the endpoint rejects
  an internal repository on a live cluster.

Result:

- A restarting replica can no longer silently skip a recovery the rest of the
  cluster applied, guaranteed-to-fail recoveries of internal repositories are
  rejected up front, and a recovery can no longer take down the cluster by
  sheer payload size.

Motivation:

A recovery requested through a non-source replica completes asynchronously
and its failure is only reported in the source replica's log, so the operator
must compare the repository head on every replica before making it writable
again. The UI explained this but left the operator to figure out how.

Modifications:

- After a REQUESTED recovery, render a copy-pastable shell script next to the
  result: one curl per replica (built from the live replica roster and the
  page origin) that prints the repository's head revision, with the source
  replica marked, plus a reminder that failures appear only in the source
  replica's log.
- Snapshot the recovered project/repository and the source server ID in the
  result state so the script always describes the recovery it belongs to.

Result:

- An administrator can verify cluster-wide convergence with a single pasted
  command block instead of hand-crafting per-replica requests.

…y UI

Motivation:

The verification script hard-coded a plain per-replica URL, which breaks on an
HTTPS deployment (the certificate and any virtual-host routing are issued for
the load balancer's name, not for a replica host). And a rejected recovery
rendered the server's whole Java stack trace, which a system administrator
always receives, burying the one sentence that explains the rejection.

Modifications:

- Keep this page's authority in each curl and dial the replica directly with
  --connect-to, so TLS validation and virtual-host routing keep working, with
  a note about `curl -k` as the alternative; default the port to Central
  Dogma's 36462 for http.
- Show only the reason of a rejected recovery: drop the stack frames and the
  exception class that repeats the message, in the inline modal error and the
  toast alike.

Result:

- The verification script is usable as-is on an HTTPS cluster, and a rejected
  recovery reads as a single actionable sentence.

Motivation:

The per-replica verification script kept the page's authority in every curl
and dialed the replica with --connect-to, which made each line long and hard
to read, and it rendered as plain unhighlighted text.

Modifications:

- Address each replica by its own host name. Over HTTPS that host is not
  covered by the certificate issued for the load balancer, so add -k there;
  over HTTP there is nothing to skip, so do not.
- Syntax-highlight the script with Prism, as the variable view already does.

Result:

- The script is a short, readable, highlighted command per replica.

Motivation:

The Recover button shared the wrapping row of the recovery form, so once the
fields wrapped it ended up next to the start revision, floating in the middle
of the second row.

Modifications:

- Give the action its own row, right-aligned, and top-align the fields so the
  inputs line up across a wrap.
- Widen the start revision field so its helper text fits on one line.

Result:

- The form reads as fields first, then the action, at every width.

Motivation:

Driving the recovery page in a browser against a two-replica cluster showed
the generated verification script seeding both replicas with the same address:

    REPLICAS='1=127.0.0.1:36462 2=127.0.0.1:36462'

The replica roster carries no HTTP port — ZooKeeper only knows the quorum
ports — so every address takes the port of the URL the page was served on.
Where replicas share a host, that collapses them onto one address. The script
then polls one server twice, gets one commit ID twice, and reads as converged
while never contacting the other replica. That is the same false pass the
script was just fixed to stop giving.

Neither the code review nor the unit tests caught it; it was only visible on
screen.

Modifications:

- Warn in the script when two replicas are seeded with the same address, and
  say what to correct before running it.
- Have the script check for itself, so the warning cannot be skimmed past: it
  reports any address it polled more than once.

Result:

The check can no longer claim a convergence it did not verify. Covered by
tests for both the collided and the distinct roster.